On September 21, 2024, at 15:14 UTC, Base Mainnet experienced a significant incident characterized by a 17-minute block building outage. This event, while disruptive, did not compromise the integrity of the blockchain, and all funds remained secure. The incident prompted Base to conduct a thorough retrospective to analyze the root cause, the impact of the outage, the mitigation steps taken, and future improvements. The outage was traced back to a misconfiguration within the sequencer cluster. When the active block producer became unhealthy, it failed to initiate block production on an alternative instance. The situation was resolved by manually restarting block production on a properly configured instance. During the outage, no blocks were produced, specifically from block numbers 20071146 to 20071691, which were generated only after block production resumed. Transaction processing continued through the `eth_sendRawTransaction` RPC call, which allows transactions to be submitted to Base and placed in the mempool. Despite the mempool functioning correctly, there was a noticeable decline in transaction submissions during the outage, likely due to applications being affected by the halt in block production. Following the resumption of block production, many transactions that had been submitted during the outage were included in the subsequent blocks. In the background, Base had been working on a system called op-conductor, designed to enhance the reliability of block production and achieve a target availability of 99.99%. Prior to implementing op-conductor, any failure of the sequencer would lead to an outage. The transition to the op-conductor cluster occurred on September 20, 2024, but the instances were misconfigured, preventing the op-node from submitting new unsafe block payloads to op-conductor. The trigger for the incident was a delay in block production experienced by the active sequencer. op-conductor detected this issue and attempted to transfer leadership to another instance. However, due to the misconfiguration, the new block producer could not start production, as it required an unsafe payload that had not been written by the previous leader. This miscommunication resulted in a state where no instance could act as an active block producer. To mitigate the incident, Base reverted to the single sequencer topology while addressing the configuration issues within the op-conductor cluster. Moving forward, Base plans to implement several improvements, including establishing a bidirectional handshake between op-node and op-conductor at startup to ensure proper communication and enhancing internal configuration management processes to prevent and detect future misconfigurations. This incident serves as a learning opportunity for Base, reinforcing its commitment to transparency and continuous improvement in its operations.

OP Mainnet introduced permissionless fault proofs, moving it into Stage 1 of decentralization with plans to extend these capabilities to more chains like Base and Metal. This upgrade enhances security by allowing any user to challenge invalid withdrawals and includes fail-safes like a Security Council that can revert the system to a permissioned state if necessary. OP Mainnet aims to eventually reach full decentralization in Stage 2.

One of the 13 root servers that provision the Internet's root zone experienced an unexplained glitch that could have caused stability and security problems worldwide.

Optimism rolled back its fraud proof system to a permissioned fallback mechanism after community-driven audits found multiple vulnerabilities. Two months ago, it reached Stage 1 rollup decentralization when anyone could contest L2 transactions on the mainnet. A proposed network upgrade, including a hard fork named "Granite," is set for September 10, aiming to enhance security without compromising user assets.

The Canto blockchain experienced a two-day outage because of a consensus issue, halting all transactions since Saturday and leading to a 21% decrease in its token price.

Bittensor developers have halted their blockchain network following the discovery of a suspected security exploit targeting users' wallets initially reported by on-chain analyst ZachXBT. The halt was enacted to prevent further unauthorized access while an investigation is conducted. Approximately $8 million worth of TAO tokens were stolen, causing a 15% drop in the token's value.

TON Network was down for 2+ hours yesterday, with major exchanges like Binance suspending withdrawals to TON.

Italian bank Sella has been offline for days after a weekend system update caused a database problem. It is working with Oracle to resolve the unexpected issue. The extended downtime shows the importance of having a good backup strategy. While "blackout" windows for maintenance are common in banking, they might inadvertently lead to weaker engineering practices compared to industries that require zero-downtime migrations.

Vitalik Buterin proposed the first block in preparation of Taiko's mainnet and expressed his excitement around Ethereum L2s taking different rollup approaches.

Midjourney has banned Stability AI staff from using its service, alleging their botnet-like activity caused a system outage by attempting to scrape data.